Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization

Preconditioning and Regularization Enable Faster Reinforcement Learning Natural policy gradient (NPG) methods, in conjunction with entropy regularization to encourage exploration, are among the most popular optimization algorithms contemporary reinforcement learning. Despite empirical success, theoretical underpinnings for NPG methods remain severely limited. In “Fast Global Convergence of Policy Gradient Methods Entropy Regularization”, Cen, Cheng, Chen, Wei, Chi develop nonasymptotic convergence guarantees entropy-regularized under softmax parameterization, focusing on tabular discounted Markov decision processes. Assuming access exact evaluation, authors demonstrate that algorithm converges linearly at an astonishing rate is independent dimension state-action space. Moreover, provably stable vis-à-vis inexactness evaluation. Accommodating a wide range learning rates, this result highlights role preconditioning enabling fast convergence.